Cite this Article
Huang Yu-Jiao, Wang Xiao-Yan, Long Hai-Xia, Yang Xu-Hua. Synthesization of high-capacity auto-associative memories using complex-valued neural networks. Chinese Physics B, 2016, 25(12): 120701  
Permissions
Synthesization of high-capacity auto-associative memories using complex-valued neural networks
Huang Yu-Jiao†, , Wang Xiao-Yan, Long Hai-Xia, Yang Xu-Hua
College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310023, China

 

† Corresponding author. E-mail: hyj0507@zjut.edu.cn

Project supported by the National Natural Science Foundation of China (Grant Nos. 61503338, 61573316, 61374152, and 11302195) and the Natural Science Foundation of Zhejiang Province, China (Grant No. LQ15F030005).

Abstract
Abstract

In this paper, a novel design procedure is proposed for synthesizing high-capacity auto-associative memories based on complex-valued neural networks with real-imaginary-type activation functions and constant delays. Stability criteria dependent on external inputs of neural networks are derived. The designed networks can retrieve the stored patterns by external inputs rather than initial conditions. The derivation can memorize the desired patterns with lower-dimensional neural networks than real-valued neural networks, and eliminate spurious equilibria of complex-valued neural networks. One numerical example is provided to show the effectiveness and superiority of the presented results.

PACS: 07.05.Mh;02.30.Ks
Keyword:associative memory;complex-valued neural network;real-imaginary-type activation function;external input
1. Introduction

Since Hopfield’s inspirational work on the stability of the Hopfield neural network using an energy function,[1] numerous studies on the dynamics of various neural networks have been reported.[2–13] Due to strong learning ability, high classification and prediction accuracy, the dynamics of neural networks have been investigated extensively to achieve associative memory and pattern recognition. Design procedures for synthesizing associative memories have been proposed based on real-valued neural networks (RVNNs)[14–20] and complexvalued neural networks (CVNNs).[21–26]

In the design procedures based on RVNNs, coexistence of many equilibria is a necessary feature in the applications of neural networks to associative memory storage or pattern recognition. Extensive investigations have been done to study the existence and stability of multiple equilibria of neural networks.[27–33] Due to the dependence on initial states of neural networks, these works resulted in the appearance of spurious equilibria. It is difficult to avoid spurious equilibria, and accurate pattern recalling cannot be guaranteed by using local stability of neural networks. In order to solve this problem, design procedures were presented for synthesizing associative memories based on RVNNs by assuring that each trajectory globally converged to a unique equilibrium point, which depended on the external inputs of neural networks, for example, see Refs. [15], [16], [19], and [20]. These designed networks defined a nonlinear mapping from the space of external inputs to that of steady state outputs. In the design procedures based on CVNNs, neural networks have been discussed as multistate associative memory models by employing a class of amplitude-phase-type activation functions, i.e., multilevel sigmoid functions. Although such dynamics are favorable for gray-level image reconstruction, it often suffers from convergence to an undesirable spurious pattern. Most works have been done to overcome this shortcoming.[21,23,24] As is well known, it is more efficient to avoid spurious patterns when the convergence of neural networks depends on external inputs instead of initial states in associative memories. However, to the best of the authors’ knowledge, less attention has been paid to designing associative memory procedures based on CVNNs dependent on external inputs.

It is well known that the activation functions play an important role in the dynamical analysis of neural networks. In RVNNs, the activation functions are usually chosen as smooth and bounded functions. However, this is not suitable in CVNNs. Since every bounded entire function must be constant in the complex domain according to Liouville’s theorem. Therefore, the choice of activation functions is the main challenge for CVNNs. Two kinds of complex-valued activation functions have been presented in previous studies, i.e., real-imaginary- type activation functions[34] and amplitude-phase-type activation functions.[23,25] Many works have been done to study the design procedure of associative memories based on CVNNs with amplitude-phase-type activation functions[21–25] and the convergence of CVNNs with real-imaginary-type activation functions.[6,34] To the best of the authors’ knowledge, less associative memory results based on CVNNs with real-imaginary-type activation functions have been published. In fact, this is a very meaningful topic. First, the real-imaginary-type activation functions are suitable for dealing with complex information that must have symmetry concerning, or a certain special meaning on the real and imaginary axes. Second, the real parts and the imaginary parts of activation functions can be some functions with multiple constant values, which is helpful in realizing the coexistence of multiple equilibrium points of the neural networks. Moreover, one of the advantages of using the real-imaginary-type activation functions is that the whole CVRNNs can be analyzed by dealing with real and imaginary parts separately and independently, which reduces the complexity of computation and analysis.

In electronic implementation of analog neural networks, time delays always exist due to the transmission of signal and the finite switching speed of amplifiers. The existence of time delays may lead to instability and oscillation in a neural network. Therefore, it is necessary to introduce delays into neural network models to synthesize the associative memories.

Motivated by the above discussions, the main goal of this paper is to investigate the design procedure of auto-associative memories based on complex-valued neural networks with real-imaginary-type activation functions and constant delays. The first main work is that we construct a class of the real-imaginary-type activation functions, which are piecewise linear functions corresponding to their real parts and imaginary parts, respectively. Secondly, four lemmas are presented to track the dynamics of solution flows of CVNNs. Based on these lemmas, global exponential stability criteria of CVNNs are established, which depend on external inputs of neural networks. Moreover, one high-capacity design procedure is proposed for synthesizing auto-associative memories. There are two difficulties on analysis and design of high-capacity auto-associative memories based on CVNNs with the real-imaginary-type activation functions.

Compared with the existing relevant results, the main advantages of this paper include the following two points.

The remainder of this paper consists of the following sections. Section 2 describes the problem formulations. The main results are presented in Section 3. Section 4 gives an example to show the effectiveness of the obtained results. Finally, in Section 5, the conclusion is drawn.

2. Preliminaries and problem formulations

Denote {−1,1}2n as the set of 2n-dimensional bipolar vectors, i.e., {−1,1}2n = {x ∈ ℝ2n, x = (x1,x2,…,x2n)T, xi = 1 or −1, i = 1,2,…,2n}. Denote i as the imaginary unit, that is . We are now in a position to address the following design problem.

Design problem Given m (m ≤ 22n) vectors α(1), α(2),…,α(m), where α(k) ∈ {−1,1}2n, k = 1,2,…,m, design an auto-associative memory based on CVNNs such that if is fed to the associative memory from its input as a probe, then the output vector of the neural network converges to corresponding pattern , k ∈ {1,2,…,m}.

Consider a complex-valued neural network with multiple delays described by

where z(t) = (z1(t), z2(t),…,zn(t))T ∈ is the neuron state vector; D = diag(d1,d2,…,dn) is a diagonal matrix, di ∈ ℝ with di > 0 represents the self-inhibition with which the i-th neuron will reset its potential to the resting state in isolations when disconnected from the network; and are the connection weight matrices those are not assumed to be symmetric; τi j is the constant delay with 0 < τi j ≤ τ (τ = max1 ≤ i,j ≤ n{τi j}); f(z(t)) = (f1(z1(t)), f2(z2(t)),…,fn(zn(t)))T: denotes the neuron activation function; stands for the external constant input.

Let , where * indicates the conjugate transpose. The initial state associated with Eq. (1) is assumed to be ϕ(s) = (ϕ1(s),ϕ2(s),…,ϕn(s))T: . Denote the norm of ϕ as

In order to synthesize the auto-associative memory in the design problem of this paper, the following hypothesis about the activation function is needed.

Hypothesis 1 Suppose the activation function can be expressed by

where

with j = 1,2,…,n.

Since the real part and imaginary part of the activation function under Hypothesis 1 have individual properties, we can separate system (1) into its real and imaginary parts. By separating the state, the connection weight, the activation function, and the external input, we have that, xi(t) and yi(t) are the real part and the imaginary part of zi(t), respectively; and are the real part and the imaginary part of ai j, respectively; and are the real part and the imaginary part of bi j, respectively; and are the real part and the imaginary part of Ii, respectively. Therefore, the system (1) can be rewritten as follows:

The definition of the exponential stability is given as follows.

Definition 1 The equilibrium point z̄ of the system (1) is said to be globally exponentially stable, if there exist constants α > 0, β > 0 such that for any t ≥ 0,

where z(t;ϕ) is the solution of system (1) with any initial condition ϕ(s), s ∈ [−τ,0].

In order to obtain the main results, we need the following lemma.

Lemma 1 (see Ref. [35]): Let be a bounded and closed set in ℝn, and H be a mapping on complete matric space (, ‖·‖), where for any is measurement in . If and there exists a positive constant α < 1 such that for any , then there exists a unique such that H(x̄) = x̄.

3. Main results

The purpose of this paper is to design an auto-associative memory based on CVNNs. Firstly, we will introduce four lemmas according to Ref. [29]. Through these four lemmas, the dynamics of each state component of system (1) will be tracked, which depend on external inputs. Denote

Lemma 2 The state component zi(t) of system (1) will flow into the region {zi = xi + iyi|xi ∈ (−∞, −1), yi ∈ (−∞, −1)}, and stay in it afterwards, if for any i ∈ {1,2,…,n},

Lemma 3 The state component zi(t) of system (1) will flow into the region {zi = xi + iyi|xi ∈ (−∞, −1), yi ∈ (1, + ∞)}, and stay in it afterwards, if for any i ∈ {1,2,…,n},

Lemma 4 The state component zi(t) of system (1) will flow into the region {zi = xi + iyi|xi ∈ (1, + ∞), yi ∈ (−∞, −1)}, and stay in it afterwards, if for any i ∈ {1,2,…,n},

Lemma 5 The state component zi(t) of system (1) will flow into the region {zi = xi + iyi|xi ∈ (1, + ∞), yi ∈ (1, + ∞)}, and stay in it afterwards, if for any i ∈ {1,2,…,n},

The proofs of Lemmas 2–5 are similar to the proof of Lemma 2 in Ref. [29], and thus are omitted here for brevity.

Denote four index subsets as follows:

It is obvious that N1 ∩N2 ∩N3 ∩N4 = ∅. Denote

Now, we will show the existence of the globally exponentially stable equilibrium point of system (1).

Theorem 1 Suppose N1 ∪ N2 ∪ N3 ∪ N4 = {1,2,…,n}. For any i ∈ {1,2,…,n}, if

then there exists a globally exponentially stable equilibrium point for system (1), and the stable output vector belongs to {1 + i,1 − i,−1 + i,−1 − i}n.

Proof We will study the existence, uniqueness, and global exponential stability of the equilibrium point for system (1) in two steps.

Step 1 We show that there exists a unique equilibrium point for system (1). Applying Lemmas 2–5, for any initial state ϕ(s) = (ϕ1(s),ϕ2(s),…,ϕn(s))T, ∃ T > 0, when t > T, we have the following four results.

The neuron state will stay in them afterwards. That is, when the state z(t) of system (1) flows into Ω, it will not escape from Ω. Therefore, we only need to prove the existence of a unique equilibrium point with initial state ϕ(s) ∈ Ω. Let H(x) = (H1(x),H2(x),…,Hn(x))T and G(y) = (G1(y),G2(y),···,Gn(y))T, where

For any i ∈ N1, there exists a small enough constant ɛ1 > 0 such that

For any i ∈ N2, there exists a small enough constant ɛ2 > 0 such that

For any i ∈ N3, there exists a small enough constant ɛ3 > 0 such that

For any i ∈ N4, there exists a small enough constant ɛ4 > 0 such that

Consider any one small enough positive constant ɛ with 0 < ɛ ≪ min{ɛ1,ɛ2,ɛ3,ɛ4}. Denote

It is obvious that Ωɛ is a bounded and closed set of Ω. When xi ∈ [−1/ɛ,−1 −ɛ], it can be derived that

When xi ∈ [1 + ɛ, 1/ɛ], it can be derived that

Similarly, when yi ∈ [−1/ɛ,−1−ɛ], we have Gi(y) < −1−ɛ; when yi ∈ [1 + ɛ, 1/ɛ], we have Gi(y) > 1 + ɛ. Because ɛ is small enough, we get

For any

, it holds that for i = 1,2,…,n. Since we only discuss the existence of equilibrium points in Ω, when fixing i ∈ {1,2,…,n}, we have for any ,. Choose α = 0.5. We have |Hi(x(1)) − Hi(x(2))| ≤ α‖x(1) − x(2)‖. Therefore, ‖H(x(1)) − H(x(2))‖ ≤ α‖x(1) − x(2)‖. According to Lemma 1, there exists a unique fixed point x̄ such that H(x̄) = x̄. By a similar analysis, there exists a unique fixed point ȳ such that G(ȳ) = ȳ. Based on the above discussion, we can derive that y = (y1,y2,…,yn)T in Eq. (8) and x = (x1,x2,···,xn)T in Eq. (9) are allowed to be arbitrary. So, we can make y = ȳ in Eq. (8) and x = x̄ in Eq. (9). Then, z̄ = x̄ + iȳ is a unique equilibrium point of system (1) in Ωɛ. By arbitrariness of ɛ, z̄ is a unique equilibrium point of system (1) in .

Step 2 We investigate the global exponential stability of equilibrium point z̄ = x̄ + iȳ of system (1). With translation v(t) = z(t) − z̄, systems (2a) and (2b) become

where v = (v1,v2,…,vn)T. For each i ∈ {1,2,…,n}, consider the single-variable function

From Eq. (7), we can get Fi(0) > 0, and there exists a positive constant λ such that Fi(λ) > 0. Consider wi(t) = eλt|Re(vi(t))|, ui(t) = eλt|Im(vi(t))|, i = 1,2,…,n. Let δ > 1 and

Therefore, for t ∈ [−τ, 0]. We will show that for all t > 0, i = 1,2,…,n. Suppose this is not true. Then there exist k ∈ {1,2,…,n}, t1 > 0 such that for t ∈ [−τ, t1], i ≠ k, for t ∈ [−τ,t1), , and , where D+ denotes the upper right Dini-derivative. From Eqs. (10a) and (10b), we compute that

Hence,

This yields a contradiction to . So, for all t > 0, i = 1,2,…,n. By taking δ → 1+, we obtain , which leads to

for any t > 0. Therefore, z(t) converges to z̄ exponentially. Since the equilibrium point z̄ lies in Ω, it is easy to derive the stable output vector of system (1) belonging to {1 + i,1 − i,−1 + i,−1 − i}n. □

Remark 1 Among the conditions of Theorem 1, N1 ∪ N2 ∪ N3 ∪ N4 = {1,2,…,n} determines the range of the external inputs, which ensures neuron states can flow into Ω according to Lemmas 2–5. In fact, Ω is the attractive region of the globally exponentially stable equilibrium point of system (1).

Remark 2 Equilibrium states of neurons of system (1) depend on external inputs of the neural network. Therefore, equilibrium states of system (1) are different when external inputs are different.

Remark 3 According to Theorem 1, we can realize associative memories based on CVNNs. We need to use the information about a lot of pairwise vectors of desired patterns and inputs to design a recurrent neural network. When the retrieval probe is determined as the input of the neural network, the output vector will converge to the desired pattern. Based on this idea, a design process of associative memories will be put forward after the following corollaries.

For the case of system (1) without delay, i.e., bi j = 0, i, j = 1,2,…,n, denote

We can derive the following corollary.

Corollary 1 Suppose N5 ∪ N6 ∪ N7 ∪ N8 = {1,2,…,n}. For any i ∈ {1,2,…,n}, if

then there exists a globally exponentially stable equilibrium point for system (1) without delay, and the stable output vector belongs to {1 + i,1 − i,−1 + i,−1 − i}n.

In particular, when yi = 0,,,,, system (1) can be rewritten as the following RVNNs,

Denote

We can derive the following corollary.

Corollary 2 Suppose N9 ∪ N10 = {1,2,…,n}. For any i ∈ {1,2,…,n}, if

then there exists a globally exponentially stable equilibrium point for system (12), and the stable output vector belongs to {1,−1}n.

Based on the above discussion, one can obtain the following design procedure of auto-associative memory.

Design procedure

In the application of associative memory, the parameters of CVNNs are designed according to the information about pairwise vectors of desired patterns and inputs. In this paper, the input vectors are designed to be the same as the vectors of desired patterns. According to the above design procedure, the parameters D, A, B can be designed. Hence, CVNNs can be derived. Once the neural networks are designed, the parameters will not change. The stable output vectors are only relevant to the input vectors. If one input is fed as a probe, the corresponding desired pattern will be memorized. It can be noticed that the choice of the parameters of neural networks depends on some inequalities. Therefore, the parameters can be changed in a certain range, which do not affect the result of associative memories.

Remark 4 In the design procedure, the conditions are necessary. From Eq. (7) and , we can derive that equation (3) holds for and equation (4) holds for and equation (5) holds for and equation (6) holds for and . That is N1 ∪ N2 ∪ N3 ∪ N4 = {1,2,…,n}.

Remark 5 Compared with the existing design procedures of auto-associative memories based on CVNNs, there exist the following advantages for the design procedure in this paper.

Remark 6 Compared with the existing design procedures of auto-associative memories based on RVNNs, there exist the following advantages for the design procedure in this paper.

4. Illustrative examples

In this section, we will give one example to illustrate the effectiveness of our results.

Example 1 Consider two to-be-stored patterns “M” and “F”, which are represented by two (8×7)-pixel images (black pixel = −1; white pixel = 1), as shown in Fig. 1.

Fig. 1. Desired output patterns.

Since the external inputs are equal to the desired patterns in the auto-associative memory, we can give the input vectors as follows:

According to the design procedure mentioned in Section 3, we consider neural network (1) with 28 neurons, and the parameters are determined as follows for i, j = 1,2,…,28,

Let τi j = 0.1, fj(ξ) = (|ξ + 1| − |ξ−1|)/2.

When the input probe is set to be I(1), the desired pattern α(1) will be memorized. When the input probe is set to be I(2), the desired pattern α(2) will be memorized. Simulations show the effectiveness of the results with four random initial values in Figs. 2 and 3.

Fig. 2. Typical evolutions of output variables with input probe I(1) under four random initial values.
Fig. 3. Typical evolutions of output variables with input probe I(2) under four random initial values.
5. Conclusion

In this paper, a new design procedure for synthesizing auto-associative memories has been proposed based on CVNNs with real-imaginary-type activation functions and constant delays. Global exponential stability criteria have been derived. The stability results depend on external inputs of neural networks. Compared with previous results, the design method for synthesizing auto-associative memories is suitable for dealing with complex information that must have symmetry concerning, or a certain special meaning on the real and imaginary axes in the applications to associative memory, eliminates spurious memory patterns, relaxes the relationship of parameters of recurrent neural networks, and has high storage capacity of auto-associative memories.

Several extensions would be welcome as future work:

Reference
1Hopfield J J 1982 Proc. Nat. Acad. Sci. 79 2554
2Hirose A2003Complex-valued Neural Networks: Theories and ApplicationsSingaporeWorld Scientific
3Zhang H GLiu Z WHuang G BWang Z S 2010 IEEE Trans. Neural Netw. 21 91
4Zhang H GYang F SLiu X DZhang Q L 2013 IEEE Trans. Neural Netw. Learn. Syst. 24 513
5Zhang H GWang J YWang Z SLiang H J 2015 IEEE Trans. Neural Netw. Learn. Syst. 26 2621
6Rao VMurthy G 2008 Int. J. Neural Syst. 18 165
7Kwon O MKwon J WKim S H 2011 Chin. Phys. 20 050505
8Song QZhao ZLiu Y 2015 Neurocomputing 168 1044
9Gong WLiang JCao J 2015 Neurocomputing 168 135
10Wen S PZeng Z GHuang T WZhang Y D 2014 IEEE Trans. Fuzzy Syst. 22 1704
11Wen S PZeng Z GHuang T WMeng Q GYao W 2015 IEEE Trans. Neural Netw. Learn. Syst. 26 1493
12Cao Y TWen S PChen Michael Z QHuang T WZeng Z G 2016 Neural Netw. 81 52
13Li NCao J 2016 Int. J. Syst. Sci. 47 2827
14Bao GZeng Z G 2012 Neurocomputing 77 101
15Grassi G 1997 IEEE Trans. Circuits Syst. I 44 835
16Grassi G 2001 IEEE Trans. Circuits Syst. I 48 107
17Han QLiao XHuang TPeng JLi CHuang H 2012 Neurocomputing 97 192
18Liu DMichel A N 1994 IEEE Trans. Circuits Syst. II 41 295
19Zeng Z GWang J 2008 IEEE Trans. Syst., Man, Cybern. 38 1525
20Zeng Z GWang J 2009 Neural Netw. 22 651
21Aizenberg N NAizenberg I N1992Proc. 2nd Int. Workshop on Cellular Neural Netw. Appl.36
22Jankowski SLozowski AZurada J M 1996 IEEE Trans. Neural Netw. 7 1491
23Lee D L 2006 IEEE Trans. Neural Netw. 17 1341
24Müezzinoǧlu M KGüeliş CZurada J M 2003 IEEE Trans. Neural Netw. 14 891
25Tanaka GAihara K 2008 IEEE Trans. Neural Netw. 20 1463
26Zurada J MCloete IVan der Poel E 1996 Neurocomputing 13 135
27Cao J DFeng GWang Y Y 2008 Physica 237 1734
28Huang ZSong QFeng C 2010 IEEE Trans. Circuits Syst. I 57 2144
29Wang L LChen T P 2012 IEEE Trans. Neural Netw. Learn. Syst. 23 1816
30Wang L LChen T P 2014 Neural Netw. 53 109
31Huang Y JZhang H GWang Z S 2014 Appl. Math. Comput. 229 187
32Zeng Z GZheng W X 2013 IEEE Trans. Neural Netw. Learn. Syst. 24 1749
33Liu PZeng Z GWang J 2016 IEEE Trans. Syst. Man. Cybernetics 46 512
34Hu JWang J 2012 IEEE Trans. Neural Netw. Learn. Syst. 23 853
35Yosida K1978Functional AnalysisBerlinSpringer-Verlag
36Hirose A 2011 Front. Electr. Electron. Eng. China 6 171